Computing devices continue to become ubiquitous to daily life. They take the form of computer desktops, laptop computers, tablet computers, hybrid computers (2-in-1s), e-book readers, mobile phones, smartphones, wearable computers, global positioning system (GPS) units, enterprise digital assistants (EDAs), personal digital assistants (PDAs), game consoles, and the like. Further, computing devices are being incorporated into vehicles and equipment, such as cars, trucks, farm equipment, manufacturing equipment, building environment control (e.g., lighting, HVAC), and home and commercial appliances.
Computing devices generally consist of at least one processing element, such as a central processing unit (CPU), some form of memory, and input and output devices. The variety of computing devices and their subsequent uses necessitate a variety of interfaces and input devices. One such input device is a touch sensitive surface such as a touch screen or touch pad wherein user input is received through contact between the user's finger or an instrument such as a pen or stylus and the touch sensitive surface. Another input device is an input interface that senses gestures made by a user above the input interface. A further input device is a position detection system which detects the relative position of either touch or non-touch interactions with a non-touch physical or virtual surface. Any of these methods of input can be used generally for the input of handwritten content. The user's handwriting is interpreted using a handwriting recognition system or method.
One application of handwriting recognition in computing devices is in the creation of diagrams which are hand-drawn on a computing device to be converted into typeset versions. Diagrams are drawings that explain or show arrangement and relations (as of parts). Diagrams generally include shapes having arbitrary or specific meanings and text with relationships to these shapes. There are many type of diagrams, such as flowcharts, organizational charts, concept maps, spider maps, block/architecture diagrams, mind-maps, block diagrams, Venn diagrams and pyramids, to name but a few.
Generally, diagrams are created to have shape elements containing text. In conventional digital (e.g., non-handwriting) diagramming applications, the size of the typeset text (e.g., the font size) within the shape containers is governed by pre-set parameters. For example, in MICROSOFT® POWERPOINT®, GOOGLE® Drawings and GOOGLE® Slides settings can be made by users to wrap text within shapes, reduce the font size when the shape size is reduced, or resize the shape when overflowing text is input. It is problematic however to apply such operations on the text and shapes of handwritten diagrams.
This is because, with respect to the digital ink (i.e., the rendered visualization of the input handwriting) version of the diagram it is difficult to resize the text and non-text in digital ink without distorting the handwriting, which adversely impacts on user experience and on the content itself since additional user effort is required with possible reduction in quality. Further, if such operations are not carried out on the digital ink but are carried out for the typeset content upon conversion of the digital ink, the resulting typeset version of the diagram may be quite different to the originally intended diagram.
Further issues exist with respect to hierarchical structures (e.g., trees) within diagrams. That is, many diagram types have levels of information, such as organizational charts, mind-maps, etc., in which each level of information is displayed in a way which visually represents the level order. In such cases, with respect to the typeset version of the diagram the different levels of the hierarchy are easily rendered with text sizes and shape sizes that are specific for the level through pre- or post-selection by users, the use of style templates and the like, or interpretation of the content.
However, representing such hierarchical levels in digital ink is difficult because as the Applicant has found, the handwriting of users without and within such different levels varies to a great degree (for example, it has been noted that users cramp handwriting to fit within already drawn shapes), such that any intended differentiation in element size is difficult to interpret through the handwriting recognition process itself. That is, if the height (or size) of the digital ink is mapped directly to font size when typesetting, the converted content may have many different font sizes for similar blocks (levels) and perhaps reversed font sizes for different levels, e.g., higher levels have smaller font size than lower levels. This can be corrected post-typesetting through intervention by users, e.g., via block or menu selection, however this reduces user experience and productivity. Alternatively, a default or standard font size can be applied across all levels or individually for each level regardless of the handwritten text size.
Another application of handwriting recognition in computing devices is in the creation of tables, and the input of content and data into tables, or other structured content, like lists. Like diagrams, tables are created to have ‘containers’ corresponding to bordered or non-bordered columns and rows defining cells containing text and may have hierarchical structure, such as column and line headings which are to have font sizes or other styling differences to the body of the table. As such, like diagrams, interpreting sensible text and non-text sizing of the typeset version of the handwritten table or list input is difficult when considering the input itself.